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^•W.cation/Control Number. 09/395.993 
Art Unit 2623 

DETAILED ACTION 
Response to Amendment 

ndment filed on October 30, 2002. has been entered and made 
1 Applicant's amendment tiiea on 

of record. , „f Annlicanfs amendment. 

■ . to Claims 4-9 has been withdrawn .nv,ew of Applicants 

. „ ,o,„eotvpes) to the classified objects (Page bUB, 
expressions (Stereotypes) 

P.ragrapt,st-3;Figures1,6(a>6(oV.Pa9e413,Coiumnt,Tabies 

on adlacencv grapt, is Cassifled based on higber-ieve, expression or 
Tbe region adjacency P ^ ^^^^^^^^ 

stereotypes (e.g., rurai road scenes). Rural 

v.no rPaae 606 Section I, second Paragrapn;- 
.sedonbigber.,eve,express,os(Pa . ^^^^^^^ 

be further emphasized that the language of Claim 
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limitations referred to in the last line of Page 17 and Page 18, Lines 1-8 of the instant 
amendment. 

Regarding Applicant's argument concerning Claim 8, it is submitted that Figure 
1(c) represents the nodes of stereotype in a hierarchical arrangement based on the 
number of neighboring regions. 

Claim Rejections - 35 (JSC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
states. 

5. Claims 1. 3-10, 18, 20-27, 35, 37-44, 52. 54-61, 66, 68-75 and 80, 82-89 are 
rejected under 35 U.S.C. 102(b) as being anticipated by Modestino et al (IEEE Paper 
ISBN: 0162-8828; A Markov random Field Model-Based Approach to Image 
Interpretation). 

Regarding Claim 1 , Modestino et al disclose a method of classifying a digital 
image, said method comprising the steps of: 

segmenting said image into substantially homogeneous regions (Figure 1; Pages 
607-608, Section ll-B. The image in figure la is segmented into homogeneous regions. 
Sky, road, Car and Fields.); 

processing said regions to provide a region adjacency graph for the digital image, 
said region adjacency graph representing adjacencies between said regions (Figure 1 ; 
Page 607, Column 2, Section MB, first Paragraph. G = {R, £} is the region adjacency 
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graph comprising of the set of nodes R = {Ri, R2, . . Rn} and the set of edges E 
connecting the regions.); 

analyzing said region adjacency graph for predetermined patterns of regions 
(Figure 1; Page 607, Column 2, Section IIB, second Paragraph. L = {Li, L2, Lm} is 
the set of all predetermined patterns of regions.); and 

classifying said digital image as one of a plurality of stereotypes according to 
each identified pattern (Figure 1; Figures 6(a)-6(c); Page 413, Column 1; Tables III (b) 
and III (c). The region adjacency graph is classified based on higher-level expression or 
stereotypes (e.g., rural road scenes). The digital image is classified based on the 
identified patterns (features) of the regions as depicted in Figure 6 and Table IV.). 

Regarding Claim 3, Modestino et al further disclose the method according to 
Claim 1 , wherein said digital image is classified on the basis of a size of one or more 
regions of said digital image (Tables l-IV. Classification is based on the areas and the 
boundary lengths of the regions identified by the RAG nodes.). 

Regarding Claim 4, Modestino et al further disclose the method according to 
Claim 3, wherein said digital image is classified on the basis of an adjacency of said 
regions (Figures 1 and 6(b). Region adjacency graphs are inherently generated based 
on the spatial dependence of objects in the relative proximity of each other as depicted 
in Figure 1(c).). 

Regarding Claim 5, Modestino et al further disclose the method according to 
Claim 1. wherein said digital image is classified on the basis of semantic label content of 
said region adjacency graph (Page 613, Table lll(b). Stereotypes are assigned to 
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region adjacency graph based on the semantic label content, i.e., "road, field" or "sky, 
field" as depicted in Table lll(b).). 

Regarding Claim 6, Modestino et al further disclose the method according to 
Claim 1 , wherein said digital image is classified on the basis of a mean color of one or 
more regions of said region adjacency graph (Page 613, Table lll(a). Average Gray 
Level of the regions). 

Regarding Claim 7, Modestino et al further disclose the method according to 
Claim 1, wherein said plurality of stereotypes are stored in an association lookup table 
(Page 607, Column 2, Section IIB, second Paragraph. Set of all interpretation labels L 
is the database or lookup table for stereotypes or node labels.). 

Regarding Claim 8, Modestino et al further disclose the method according to 
Claim 1, wherein said stereotypes are represented in a hierarchal arrangement (Figure 
1; Tables 1(a) and 19b). Figure 1(c) represents the nodes of stereotype in a hierarchical 
arrangement based on the number of neighboring regions. Based on associated 
cliques, Sky, Road and Car are represented in a hierarchical arrangement.). 

Regarding Claim 9, Modestino et al further disclose the method according to 
Claim 7, wherein each of said stereotypes has a hierarchical path arrangement (Figure 
1. Sky, Road and Car has a hierarchical path arrangement. Same concept is 
applicable to the higher-level expressions or node labels in Tables lll(b) and IV(b).). 

Regarding Claim 10, Modestino et al disclose the method according to claim 1, 
wherein said region adjacency graph is provided by analyzing contextual data 
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associated with one or more regions of said digital image (Figure 1 , Page 607, Section 
MB, first Paragraph). 

With regards to Claims 18, 35, 52, 66 and 80, arguments analogous to those 
presented for Claim 1 are applicable to Claims 18, 35, 52, 66 and 80. Concerning 
Claims 52, 66 and 80, Modestino et al further disclose providing a set of labeled regions 
(Abstract Lines 5-10; Page 607, Column 2, Section MB, second paragraph, set of labels 
L = {Li, L2, Lm}). Markov Random Field (MRF) model-based approach segments the 
image into a collection of disjoint regions that form the nodes of an adjacency graph. 
Once the adjacency graph has been determined, image classification will be achieved 
through assigning object labels to the segmented regions using domain knowledge, 
extracted feature measurements, and spatial relationship between the various regions 
(Abstract, Lines 5-10). 

With regards to Claims 20, 37, 56, 70 and 84, arguments analogous to those 
presented for Claim 3 are applicable to Claims 20, 37, 56, 70 and 84. 

With regards to Claims 21 , 38, 55, 69 and 83, arguments analogous to those 
presented for Claim 4 are applicable to Claims 21 , 38, 55, 69 and 83. 

With regards to Claims 22, 39, 54, 68 and 82, arguments analogous to those 
presented for Claim 5 are applicable to Claims 22, 39, 54, 68 and 82. 

With regards to Claims 23, 40, 57, 58, 71 , 72, 85 and 86, arguments analogous 
to those presented for Claim 6 are applicable to Claims 23, 40, 57, 58, 71 , 72, 85 and 
86. 



Application/Control Number: 09/395,993 Page 7 

Art Unit: 2623 

With regards to Claims 24, 41 , 59, 73 and 87, arguments analogous to those 
presented for Claim 7 are applicable to Claims 24, 41, 59, 73 and 87. 

With regards to Claims 25, 42, 60, 74 and 88, arguments analogous to those 
presented for Claim 8 are applicable to Claims 25, 42, 60, 74 and 88. 

With regards to Claims 26, 43, 61 , 75 and 89, arguments analogous to those 
presented for Claim 9 are applicable to Claims 26, 43, 61, 75 and 89. 

With regards to Claims 27 and 44, arguments analogous to those presented for 
Claim 10 are applicable to Claims 27 and 44. 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill In the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1 1 , 17, 28, 34, 45, 62-65, 76-79 and 90-93 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Modestino et al in view of Li et a! (U.S. 
5,930,783). 

Regarding Claim 11, Modestino et al do not explicitly disclose the method 
according to Claim 10, wherein said contextual data comprises information generated 
by one or more separate sources of said information. 

Li et al disclose a semantic and cognition based image retrieval methodology 
comprising analyzing contextual data generated by one or more separate sources of 
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information (Figure 1B, Semantic-based Query, Cognition-based Query; Column 12, 
Lines 30-40). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to analyze contextual data generated by one or more separate sources of 
information because it will expand versatility of the image segmentation and 
classification. It will utilize the advantages of image retrieval based on both image 
semantics and visual examples of the image (Li et al, Column 3, Lines 27-35). 

Regarding Claim 17, Modestino et al do not explicitly disclose the method 
according to Claim 1, wherein said digital image is stored in a database of digital 
images and wherein said classification can be used to retrieve said digital image from 
said database. 

Li et al disclose a semantic and cognition based image retrieval methodology 
wherein the digital image is stored in a database of digital images and wherein the 
classification can be used to retrieve said digital image from the database (Column 4, 
Lines 32-50). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of Li 
et al to use classification to retrieve the digital images stored in a database because it 
will utilize a plurality of information sources for image retrieval including semantic- 
based, cognition-based and scene-based queries thereby eliminating weakness of the 
individual approaches (Li et al; Column 3, Lines 27-35). 
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With regards to Claims 28 and 45, arguments analogous to those presented for 
Claim 1 1 are applicable to Claims 28 and 45. 

With regards to Claim 34, arguments analogous to those presented for Claim 17 
are applicable to Claim 34. 

With regards to Claims 62, 76 and 90, arguments analogous to those 
represented for Claim 64 concerning utilizing an icon is applicable to claims 62, 76 and 
90. 

With regards to Claims 63, 77 and 91, arguments analogous to those presented 
for Claim 17 are applicable to Claims 63, 77 and 91. Li et al retrieve the digital image 
by using a keyword representing a stereotype (Column 12, Lines 30-50). 

With regards to Claims 64, 78 and 92, arguments analogous to those presented 
for Claim 17 are applicable to Claims 64, 78 and 92. Neither Modestino nor Li et al 
explicitly retrieve the digital image by using an icon to represent a stereotype. Utilizing 
an icon to represent a keyword is extremely well known in the art (Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination to Utilize an icon 
for representing a keyword because icons are a significant factor in the user- 
friendliness of graphical user interface that serve as visual mnemonics to allow the user 
to control certain computer actions without having to remember commands or type them 
at the keyboard. 

With regards to Claims 65, 79 and 93, arguments analogous to those presented 
for Claim 17 are applicable to Claims 65, 79 and 93. Li et al retrieve the digital image 
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by using a keyword representing a generalization of a stereotype (Column 12, Lines 30- 
50). 

8. Claims 12-16, 29 and 46-51 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Modestino et al further in view of Li et al (U.S. 5,930,783) and Jain et 
al (US 6,360,234). 

Regarding Claim 12, neither Modestino et al nor Li et al disclose the method 
according to Claim 1 1 , wherein a corresponding portion of said contextual data is 
obtained from a temporal region of interest for each source of said information. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image wherein a corresponding portion of contextual data is 
obtained from a temporal region of interest for each source of information (Figures 6-9; 
Column 6, Lines 30-67). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination according to the 
teachings of Jain et al to obtain a corresponding portion of the contextual data from a 
temporal region of interest for each source of information because it will accurately 
classify and intelligently extract information, termed metadata, about the contents of 
video stream in real time (Jain et al. Column 1 , Lines 46-49). 

Regarding Claim 13, Jain et al further disclose the method according to Claim 12, 
further comprising the step of providing metadata associated with said digital image, 
wherein said metadata includes said stereotypes of said digital image (Figures 16 and 
17; Column 13, Lines 52-67, Column 14, Lines 1-31). 
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Regarding Clainn 14, Jain et al further disclose the method according to Claim 13, 
wherein said metadata includes a hierarchical path associated with said respective 
stereotype of each digital image (Figures 9, 15-17; Table 1; Column 8, Lines 22-61). 

Regarding Claim 15, Jain et al further disclose the method according to Claim 14, 
wherein said hierarchical path is stored with a respective stereotype as a metadata 
object which is associated with a respective image object (Figure 6; Column 6, Lines 
29-38). 

Regarding Claim 16, Jain et al further disclose the method according to Claim 14, 
wherein said hierarchical path is stored as a referenced lookup table (Figure 7). 

With regards to Claims 29 and 46, arguments analogous to those presented for 
Claim 12 are applicable to Claims 29 and 46. 

With regards to Claim 47, arguments analogous to those presented for Claim 13 
are applicable to Claim 47. 

With regards to Claim 48, arguments analogous to those presented for Claim 14 
are applicable to Claim 48. 

With regards to Claim 49, arguments analogous to those presented for Claim 15 
are applicable to Claim 49. 

With regards to Claim 50, arguments analogous to those presented for Claim 16 
are applicable to Claim 50. 

With regards to Claim 51, arguments analogous to those presented for Claim 17 
are applicable to Claim 51. 



Application/Control Number: 09/395,993 Page 12 

Art Unit: 2623 

9. Claims 30-33 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Modestino et al in view of Jain et al (US 6,360,234). 

Regarding Claim 30, Modestino et al do not disclose the method according to 
Claim 18 further comprising the step of providing metadata associated with said digital 
image, wherein said metadata includes said stereotypes of said digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the digital image, wherein the metadata includes stereotypes of the 
digital image (Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31), 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes stereotypes of the digital image because it will accurately classify and 
intelligently extract information, termed metadata, about the contents of video stream in 
real time (Jain et al, Column 1, Lines 46-49). 

With regards to Claim 31, arguments analogous to those presented for Claim 14 
are applicable to Claim 31 . 

With regards to Claim 32, arguments analogous to those presented for Claim 15 
are applicable to Claim 32. 

With regards to Claim 33, arguments analogous to those presented for Claim 16 
are applicable to Claim 33. 
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Conclusion 



10. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action Is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 



1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Mehrdad Dastouri whose telephone number is (703) 
305-2438. The examiner can normally be reached on Monday to Friday from 8:00 a.m. 
to 4:30 p.m. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Amelia Au can be reached on (703) 308-6604. 

The fax phone numbers for the organization where this application or proceeding 
is assigned are (703) 872-9314 for regular and for After Final communications. 



Contact Information 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone nunnber is (703) 305- 
4700. 



Mehrdad Dastouri 
Patent Examiner 
Group Art Unit 2623 
January 6, 2003 



